{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0689733d",
   "metadata": {},
   "source": [
    "# Retrieval\n",
    "\n",
    "Retrieval is the centerpiece of our retrieval augmented generation (RAG) flow. \n",
    "\n",
    "Let's get our vectorDB from before."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed2569c6",
   "metadata": {},
   "source": [
    "## Vectorstore retrieval\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "51b15e5c-9b92-4d40-a149-b56335d330d9",
   "metadata": {
    "height": 166,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "import sys\n",
    "sys.path.append('../..')\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file\n",
    "\n",
    "openai.api_key  = os.environ['OPENAI_API_KEY']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c18f8a7b-62af-403e-9877-18d1c2265b4f",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#!pip install lark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2d552e1",
   "metadata": {},
   "source": [
    "### Similarity Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fe368042",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.vectorstores import Chroma\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "persist_directory = 'docs/chroma/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a0189dc5",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [],
   "source": [
    "embedding = OpenAIEmbeddings()\n",
    "vectordb = Chroma(\n",
    "    persist_directory=persist_directory,\n",
    "    embedding_function=embedding\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3659e0f7",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "209\n"
     ]
    }
   ],
   "source": [
    "print(vectordb._collection.count())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a807c758",
   "metadata": {
    "height": 114,
    "tags": []
   },
   "outputs": [],
   "source": [
    "texts = [\n",
    "    \"\"\"The Amanita phalloides has a large and imposing epigeous (aboveground) fruiting body (basidiocarp).\"\"\",\n",
    "    \"\"\"A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.\"\"\",\n",
    "    \"\"\"A. phalloides, a.k.a Death Cap, is one of the most poisonous of all known mushrooms.\"\"\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "715d54f3",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "smalldb = Chroma.from_texts(texts, embedding=embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9a37b5a5",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"Tell me about all-white mushrooms with large fruiting bodies\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "24e3b025",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.', metadata={}),\n",
       " Document(page_content='The Amanita phalloides has a large and imposing epigeous (aboveground) fruiting body (basidiocarp).', metadata={})]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "smalldb.similarity_search(question, k=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4daa6c0d",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.', metadata={}),\n",
       " Document(page_content='A. phalloides, a.k.a Death Cap, is one of the most poisonous of all known mushrooms.', metadata={})]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "smalldb.max_marginal_relevance_search(question,k=2, fetch_k=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a29e8c9",
   "metadata": {},
   "source": [
    "### Addressing Diversity: Maximum marginal relevance\n",
    "\n",
    "Last class we introduced one problem: how to enforce diversity in the search results.\n",
    " \n",
    "`Maximum marginal relevance` strives to achieve both relevance to the query *and diversity* among the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9bb2c0a9",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"what did they say about matlab?\"\n",
    "docs_ss = vectordb.similarity_search(question,k=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a2670cfe",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people call it a free ve rsion of MATLAB, which it sort  of is, sort of isn\\'t.  \\nSo I guess for those of you that haven\\'t s een MATLAB before, and I know most of you \\nhave, MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to \\nplot data. And it\\'s sort of an extremely easy to  learn tool to use for implementing a lot of \\nlearning algorithms.  \\nAnd in case some of you want to work on your  own home computer or something if you \\ndon\\'t have a MATLAB license, for the purposes of  this class, there\\'s also — [inaudible] \\nwrite that down [inaudible] MATLAB — there\\' s also a software package called Octave \\nthat you can download for free off the Internet. And it has somewhat fewer features than MATLAB, but it\\'s free, and for the purposes of  this class, it will work for just about \\neverything.  \\nSo actually I, well, so yeah, just a side comment for those of you that haven\\'t seen \\nMATLAB before I guess, once a colleague of mine at a different university, not at \\nStanford, actually teaches another machine l earning course. He\\'s taught it for many years. \\nSo one day, he was in his office, and an old student of his from, lik e, ten years ago came \\ninto his office and he said, \"Oh, professo r, professor, thank you so much for your', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 8}),\n",
       " Document(page_content='those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people call it a free ve rsion of MATLAB, which it sort  of is, sort of isn\\'t.  \\nSo I guess for those of you that haven\\'t s een MATLAB before, and I know most of you \\nhave, MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to \\nplot data. And it\\'s sort of an extremely easy to  learn tool to use for implementing a lot of \\nlearning algorithms.  \\nAnd in case some of you want to work on your  own home computer or something if you \\ndon\\'t have a MATLAB license, for the purposes of  this class, there\\'s also — [inaudible] \\nwrite that down [inaudible] MATLAB — there\\' s also a software package called Octave \\nthat you can download for free off the Internet. And it has somewhat fewer features than MATLAB, but it\\'s free, and for the purposes of  this class, it will work for just about \\neverything.  \\nSo actually I, well, so yeah, just a side comment for those of you that haven\\'t seen \\nMATLAB before I guess, once a colleague of mine at a different university, not at \\nStanford, actually teaches another machine l earning course. He\\'s taught it for many years. \\nSo one day, he was in his office, and an old student of his from, lik e, ten years ago came \\ninto his office and he said, \"Oh, professo r, professor, thank you so much for your', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 8}),\n",
       " Document(page_content='into his office and he said, \"Oh, professo r, professor, thank you so much for your \\nmachine learning class. I learned so much from it. There\\'s this stuff that I learned in your \\nclass, and I now use every day. And it\\'s help ed me make lots of money, and here\\'s a \\npicture of my big house.\"  \\nSo my friend was very excited. He said, \"W ow. That\\'s great. I\\'m glad to hear this \\nmachine learning stuff was actually useful. So what was it that you learned? Was it \\nlogistic regression? Was it the PCA? Was it the data ne tworks? What was it that you \\nlearned that was so helpful?\" And the student said, \"Oh, it was the MATLAB.\"  \\nSo for those of you that don\\'t know MATLAB yet, I hope you do learn it. It\\'s not hard, \\nand we\\'ll actually have a short MATLAB tutori al in one of the discussion sections for \\nthose of you that don\\'t know it.  \\nOkay. The very last piece of logistical th ing is the discussion s ections. So discussion \\nsections will be taught by the TAs, and atte ndance at discussion sections is optional, \\nalthough they\\'ll also be recorded and televi sed. And we\\'ll use the discussion sections \\nmainly for two things. For the next two or th ree weeks, we\\'ll use the discussion sections \\nto go over the prerequisites to this class or if some of you haven\\'t seen probability or \\nstatistics for a while or maybe algebra, we\\'ll go over those in the discussion sections as a \\nrefresher for those of you that want one.', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 8})]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs_ss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f07f8793",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people '"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs_ss[0].page_content[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e9f7e165",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people '"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs_ss[1].page_content[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c4ca1b6",
   "metadata": {},
   "source": [
    "Note the difference in results with `MMR`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "222234c5",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people call it a free ve rsion of MATLAB, which it sort  of is, sort of isn\\'t.  \\nSo I guess for those of you that haven\\'t s een MATLAB before, and I know most of you \\nhave, MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to \\nplot data. And it\\'s sort of an extremely easy to  learn tool to use for implementing a lot of \\nlearning algorithms.  \\nAnd in case some of you want to work on your  own home computer or something if you \\ndon\\'t have a MATLAB license, for the purposes of  this class, there\\'s also — [inaudible] \\nwrite that down [inaudible] MATLAB — there\\' s also a software package called Octave \\nthat you can download for free off the Internet. And it has somewhat fewer features than MATLAB, but it\\'s free, and for the purposes of  this class, it will work for just about \\neverything.  \\nSo actually I, well, so yeah, just a side comment for those of you that haven\\'t seen \\nMATLAB before I guess, once a colleague of mine at a different university, not at \\nStanford, actually teaches another machine l earning course. He\\'s taught it for many years. \\nSo one day, he was in his office, and an old student of his from, lik e, ten years ago came \\ninto his office and he said, \"Oh, professo r, professor, thank you so much for your', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 8}),\n",
       " Document(page_content='algorithm then? So what’s different? How come  I was making all that noise earlier about \\nleast squares regression being a bad idea for classification problems and then I did a \\nbunch of math and I skipped some steps, but I’m, sort of, claiming at the end they’re \\nreally the same learning algorithm?  \\nStudent: [Inaudible] constants?  \\nInstructor (Andrew Ng) :Say that again.  \\nStudent: [Inaudible]  \\nInstructor (Andrew Ng) :Oh, right. Okay, cool.', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 13}),\n",
       " Document(page_content=\"learning algorithms to teach a car how to  drive at reasonably high speeds off roads \\navoiding obstacles.  \\nAnd on the lower right, that's a robot program med by PhD student Eva Roshen to teach a \\nsort of somewhat strangely configured robot how to get on top of an obstacle, how to get \\nover an obstacle. Sorry. I know the video's kind of small. I hope you can sort of see it. \\nOkay?  \\nSo I think all of these are robots that I thi nk are very difficult to hand-code a controller \\nfor by learning these sorts of l earning algorithms. You can in relatively short order get a \\nrobot to do often pretty amazing things.  \\nOkay. So that was most of what I wanted to say today. Just a couple more last things, but \\nlet me just check what questions you have righ t now. So if there are no questions, I'll just \\nclose with two reminders, which are after class today or as you start to talk with other \\npeople in this class, I just encourage you again to start to form project partners, to try to \\nfind project partners to do your project with. And also, this is a good time to start forming \\nstudy groups, so either talk to your friends  or post in the newsgroup, but we just \\nencourage you to try to star t to do both of those today, okay? Form study groups, and try \\nto find two other project partners.  \\nSo thank you. I'm looking forward to teaching this class, and I'll see you in a couple of \\ndays.\", metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 20})]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs_mmr = vectordb.max_marginal_relevance_search(question,k=3)\n",
    "docs_mmr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93b20226",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs_mmr[0].page_content[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17d39762",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs_mmr[1].page_content[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2b909bc",
   "metadata": {},
   "source": [
    "### Addressing Specificity: working with metadata\n",
    "\n",
    "In last lecture, we showed that a question about the third lecture can include results from other lectures as well.\n",
    "\n",
    "To address this, many vectorstores support operations on `metadata`.\n",
    "\n",
    "`metadata` provides context for each embedded chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3c1a60b2",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"what did they say about regression in the third lecture?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a8612840",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs = vectordb.similarity_search(\n",
    "    question,\n",
    "    k=3,\n",
    "    filter={\"source\":\"docs/cs229_lectures/MachineLearning-Lecture03.pdf\"}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "721f128b",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Student: It’s the lowest it –  \\nInstructor (Andrew Ng) :No, exactly. Right. So zero to the same, this is not the same, \\nright? And the reason is, in logi stic regression this is diffe rent from before, right? The \\ndefinition of this H subscript theta of XI is not the same as the definition I was using in \\nthe previous lecture. And in pa rticular this is no longer thet a transpose XI. This is not a \\nlinear function anymore. This is  a logistic function of theta transpose XI. Okay? So even \\nthough this looks cosmetically similar, even though this is similar on the surface, to the \\nBastrian descent rule I derive d last time for least squares regression this is actually a \\ntotally different learning algorithm. Okay? And it turns out that there’s actually no \\ncoincidence that you ended up with the same l earning rule. We’ll actually talk a bit more \\nabout this later when we talk about generalized linear models. But this is one of the most \\nelegant generalized learning models that we’l l see later. That even though we’re using a \\ndifferent model, you actually ended up with wh at looks like the sa me learning algorithm \\nand it’s actually no coincidence. Cool.  \\nOne last comment as part of a sort of l earning process, over here I said I take the \\nderivatives and I ended up with this line . I didn’t want to make you sit through a long \\nalgebraic derivation, but later t oday or later this week, pleas e, do go home and look at our', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 14}),\n",
       " Document(page_content='MachineLearning-Lecture03  \\nInstructor (Andrew Ng) :Okay. Good morning and welcome b ack to the third lecture of \\nthis class. So here’s what I want to do t oday, and some of the topics I do today may seem \\na little bit like I’m jumping, sort  of, from topic to topic, but here’s, sort of, the outline for \\ntoday and the illogical flow of ideas. In the last lecture, we  talked about linear regression \\nand today I want to talk about sort of an  adaptation of that called locally weighted \\nregression. It’s very a popular  algorithm that’s actually one of my former mentors \\nprobably favorite machine learning algorithm.  \\nWe’ll then talk about a probabl e second interpretation of linear regression and use that to \\nmove onto our first classification algorithm, which is logistic regr ession; take a brief \\ndigression to tell you about something cal led the perceptron algorithm, which is \\nsomething we’ll come back to, again, later this  quarter; and time allowing I hope to get to \\nNewton’s method, which is an algorithm fo r fitting logistic regression models.  \\nSo this is recap where we’re talking about in the previous lecture, remember the notation \\nI defined was that I used this X superscrip t I, Y superscript I to denote the I training \\nexample. And when we’re talking about linear regression or linear l east squares, we use \\nthis to denote the predicted value of “by my hypothesis H” on the input XI. And my', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 0}),\n",
       " Document(page_content='Instructor (Andrew Ng) :Yeah, yeah. I mean, you’re asking about overfitting, whether \\nthis is a good model. I thi nk let’s – the thing’s you’re mentioning are maybe deeper \\nquestions about learning algorithms  that we’ll just come back to later, so don’t really \\nwant to get into that right now. Any more questions? Okay.  \\nSo this endows linear regression with a proba bilistic interpretati on. I’m actually going to \\nuse this probabil – use this, sort of, probabilist ic interpretation in order to derive our next \\nlearning algorithm, which will be our first classification algorithm. Okay? So you’ll recall \\nthat I said that regression problems are where the variable Y that you’re trying to predict \\nis continuous values. Now I’m actually gonna ta lk about our first cl assification problem, \\nwhere the value Y you’re trying to predict will be discreet value. You can take on only a \\nsmall number of discrete values and in th is case I’ll talk about binding classification \\nwhere Y takes on only two values, right? So you  come up with classi fication problems if \\nyou’re trying to do, say, a medical diagnosis and try to decide based on some features that \\nthe patient has a disease or does not have a di sease. Or if in the housing example, maybe \\nyou’re trying to decide will this house sell in the next six months or not and the answer is \\neither yes or no. It’ll either be  sold in the next six months or it won’t be. Other standing', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10}),\n",
       " Document(page_content='answer. You predict that if X is to the right of, sort of, the mid-point here then Y is one \\nand then next to the left of that mid-point then Y is zero.  \\nSo some people actually do this. Apply linear  regression to classi fication problems and \\nsometimes it’ll work okay, but in general it’s actually a pretty bad idea to apply linear \\nregression to classification problems like thes e and here’s why. Let’s say I change my \\ntraining set by giving you just one more tr aining example all the way up there, right? \\nImagine if given this training set is actually  still entirely obvious  what the relationship \\nbetween X and Y is, right? It’s ju st – take this value as greate r than Y is one and it’s less \\nthen Y is zero. By giving you this additiona l training example it really shouldn’t change \\nanything. I mean, I didn’t really convey much  new information. There’s no surprise that \\nthis corresponds to Y equals one. But if you now  fit linear regression to this data set you \\nend up with a line that, I don’t know, maybe  looks like that, right? And now the', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10})]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "97031876",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 0}\n",
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 14}\n",
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 4}\n"
     ]
    }
   ],
   "source": [
    "for d in docs:\n",
    "    print(d.metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2708f6ae",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ccc2d784",
   "metadata": {},
   "source": [
    "### Addressing Specificity: working with metadata using self-query retriever\n",
    "\n",
    "But we have an interesting challenge: we often want to infer the metadata from the query itself.\n",
    "\n",
    "To address this, we can use `SelfQueryRetriever`, which uses an LLM to extract:\n",
    " \n",
    "1. The `query` string to use for vector search\n",
    "2. A metadata filter to pass in as well\n",
    "\n",
    "Most vector databases support metadata filters, so this doesn't require any new databases or indexes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b1d06084",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.llms import OpenAI\n",
    "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
    "from langchain.chains.query_constructor.base import AttributeInfo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0aa5e698",
   "metadata": {
    "height": 233,
    "tags": []
   },
   "outputs": [],
   "source": [
    "metadata_field_info = [\n",
    "    AttributeInfo(\n",
    "        name=\"source\",\n",
    "        description=\"The lecture the chunk is from, should be one of `docs/cs229_lectures/MachineLearning-Lecture01.pdf`, `docs/cs229_lectures/MachineLearning-Lecture02.pdf`, or `docs/cs229_lectures/MachineLearning-Lecture03.pdf`\",\n",
    "        type=\"string\",\n",
    "    ),\n",
    "    AttributeInfo(\n",
    "        name=\"page\",\n",
    "        description=\"The page from the lecture\",\n",
    "        type=\"integer\",\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e143f05-908f-463d-a9de-408526c3947f",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Note:** The default model for `OpenAI` (\"from langchain.llms import OpenAI\") is `text-davinci-003`. Due to the deprication of OpenAI's model `text-davinci-003` on 4 January 2024, you'll be using OpenAI's recommended replacement model `gpt-3.5-turbo-instruct` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fc9de693-7bdb-463e-b124-9f72163b0bd8",
   "metadata": {
    "height": 166,
    "tags": []
   },
   "outputs": [],
   "source": [
    "document_content_description = \"Lecture notes\"\n",
    "llm = OpenAI(model='gpt-3.5-turbo-instruct', temperature=0)\n",
    "retriever = SelfQueryRetriever.from_llm(\n",
    "    llm,\n",
    "    vectordb,\n",
    "    document_content_description,\n",
    "    metadata_field_info,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "79d781b9",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"what did they say about regression in the third lecture?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51778b0-1fcd-40a4-bd6b-0f13fec8acb1",
   "metadata": {},
   "source": [
    "**You will receive a warning** about predict_and_parse being deprecated the first time you executing the next line. This can be safely ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1d4f9f7d",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query='regression' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='source', value='docs/cs229_lectures/MachineLearning-Lecture03.pdf') limit=None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Student: It’s the lowest it –  \\nInstructor (Andrew Ng) :No, exactly. Right. So zero to the same, this is not the same, \\nright? And the reason is, in logi stic regression this is diffe rent from before, right? The \\ndefinition of this H subscript theta of XI is not the same as the definition I was using in \\nthe previous lecture. And in pa rticular this is no longer thet a transpose XI. This is not a \\nlinear function anymore. This is  a logistic function of theta transpose XI. Okay? So even \\nthough this looks cosmetically similar, even though this is similar on the surface, to the \\nBastrian descent rule I derive d last time for least squares regression this is actually a \\ntotally different learning algorithm. Okay? And it turns out that there’s actually no \\ncoincidence that you ended up with the same l earning rule. We’ll actually talk a bit more \\nabout this later when we talk about generalized linear models. But this is one of the most \\nelegant generalized learning models that we’l l see later. That even though we’re using a \\ndifferent model, you actually ended up with wh at looks like the sa me learning algorithm \\nand it’s actually no coincidence. Cool.  \\nOne last comment as part of a sort of l earning process, over here I said I take the \\nderivatives and I ended up with this line . I didn’t want to make you sit through a long \\nalgebraic derivation, but later t oday or later this week, pleas e, do go home and look at our', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 14}),\n",
       " Document(page_content='MachineLearning-Lecture03  \\nInstructor (Andrew Ng) :Okay. Good morning and welcome b ack to the third lecture of \\nthis class. So here’s what I want to do t oday, and some of the topics I do today may seem \\na little bit like I’m jumping, sort  of, from topic to topic, but here’s, sort of, the outline for \\ntoday and the illogical flow of ideas. In the last lecture, we  talked about linear regression \\nand today I want to talk about sort of an  adaptation of that called locally weighted \\nregression. It’s very a popular  algorithm that’s actually one of my former mentors \\nprobably favorite machine learning algorithm.  \\nWe’ll then talk about a probabl e second interpretation of linear regression and use that to \\nmove onto our first classification algorithm, which is logistic regr ession; take a brief \\ndigression to tell you about something cal led the perceptron algorithm, which is \\nsomething we’ll come back to, again, later this  quarter; and time allowing I hope to get to \\nNewton’s method, which is an algorithm fo r fitting logistic regression models.  \\nSo this is recap where we’re talking about in the previous lecture, remember the notation \\nI defined was that I used this X superscrip t I, Y superscript I to denote the I training \\nexample. And when we’re talking about linear regression or linear l east squares, we use \\nthis to denote the predicted value of “by my hypothesis H” on the input XI. And my', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 0}),\n",
       " Document(page_content='Instructor (Andrew Ng) :Yeah, yeah. I mean, you’re asking about overfitting, whether \\nthis is a good model. I thi nk let’s – the thing’s you’re mentioning are maybe deeper \\nquestions about learning algorithms  that we’ll just come back to later, so don’t really \\nwant to get into that right now. Any more questions? Okay.  \\nSo this endows linear regression with a proba bilistic interpretati on. I’m actually going to \\nuse this probabil – use this, sort of, probabilist ic interpretation in order to derive our next \\nlearning algorithm, which will be our first classification algorithm. Okay? So you’ll recall \\nthat I said that regression problems are where the variable Y that you’re trying to predict \\nis continuous values. Now I’m actually gonna ta lk about our first cl assification problem, \\nwhere the value Y you’re trying to predict will be discreet value. You can take on only a \\nsmall number of discrete values and in th is case I’ll talk about binding classification \\nwhere Y takes on only two values, right? So you  come up with classi fication problems if \\nyou’re trying to do, say, a medical diagnosis and try to decide based on some features that \\nthe patient has a disease or does not have a di sease. Or if in the housing example, maybe \\nyou’re trying to decide will this house sell in the next six months or not and the answer is \\neither yes or no. It’ll either be  sold in the next six months or it won’t be. Other standing', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10}),\n",
       " Document(page_content='answer. You predict that if X is to the right of, sort of, the mid-point here then Y is one \\nand then next to the left of that mid-point then Y is zero.  \\nSo some people actually do this. Apply linear  regression to classi fication problems and \\nsometimes it’ll work okay, but in general it’s actually a pretty bad idea to apply linear \\nregression to classification problems like thes e and here’s why. Let’s say I change my \\ntraining set by giving you just one more tr aining example all the way up there, right? \\nImagine if given this training set is actually  still entirely obvious  what the relationship \\nbetween X and Y is, right? It’s ju st – take this value as greate r than Y is one and it’s less \\nthen Y is zero. By giving you this additiona l training example it really shouldn’t change \\nanything. I mean, I didn’t really convey much  new information. There’s no surprise that \\nthis corresponds to Y equals one. But if you now  fit linear regression to this data set you \\nend up with a line that, I don’t know, maybe  looks like that, right? And now the', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10})]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = retriever.get_relevant_documents(question)\n",
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db04374e",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "for d in docs:\n",
    "    print(d.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "297b8168",
   "metadata": {},
   "source": [
    "### Additional tricks: compression\n",
    "\n",
    "Another approach for improving the quality of retrieved docs is compression.\n",
    "\n",
    "Information most relevant to a query may be buried in a document with a lot of irrelevant text. \n",
    "\n",
    "Passing that full document through your application can lead to more expensive LLM calls and poorer responses.\n",
    "\n",
    "Contextual compression is meant to fix this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a060cf74",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.retrievers import ContextualCompressionRetriever\n",
    "from langchain.retrievers.document_compressors import LLMChainExtractor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "038649c8",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "def pretty_print_docs(docs):\n",
    "    print(f\"\\n{'-' * 100}\\n\".join([f\"Document {i+1}:\\n\\n\" + d.page_content for i, d in enumerate(docs)]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fc686cf2",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LLMChainExtractor(llm_chain=LLMChain(memory=None, callbacks=None, callback_manager=None, verbose=False, tags=None, prompt=PromptTemplate(input_variables=['question', 'context'], output_parser=NoOutputParser(no_output_str='NO_OUTPUT'), partial_variables={}, template='Given the following question and context, extract any part of the context *AS IS* that is relevant to answer the question. If none of the context is relevant return NO_OUTPUT. \\n\\nRemember, *DO NOT* edit the extracted parts of the context.\\n\\n> Question: {question}\\n> Context:\\n>>>\\n{context}\\n>>>\\nExtracted relevant parts:', template_format='f-string', validate_template=True), llm=OpenAI(cache=None, verbose=False, callbacks=None, callback_manager=None, tags=None, client=<class 'openai.api_resources.completion.Completion'>, model_name='gpt-3.5-turbo-instruct', temperature=0.0, max_tokens=256, top_p=1, frequency_penalty=0, presence_penalty=0, n=1, best_of=1, model_kwargs={}, openai_api_key='eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcHAiLCJleHAiOjE3OTk5OTk5OTksInN1YiI6MjQ3OTQzNCwiYXVkIjoiV0VCIiwiaWF0IjoxNjk0MDc2ODUxfQ.O_hJvIJ6Rs5hzlR3OSO-Z1A_fNkCI6Fs3ZFS7G_pbaw', openai_api_base='http://jupyter-api-proxy.internal.dlai/rev-proxy/langchain', openai_organization='', openai_proxy='', batch_size=20, request_timeout=None, logit_bias={}, max_retries=6, streaming=False, allowed_special=set(), disallowed_special='all'), output_key='text', output_parser=NoOpOutputParser(), return_final_only=True, llm_kwargs={}), get_input=<function default_get_input at 0x7f644de3aa60>)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Wrap our vectorstore\n",
    "llm = OpenAI(temperature=0, model=\"gpt-3.5-turbo-instruct\")\n",
    "compressor = LLMChainExtractor.from_llm(llm)\n",
    "compressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "82794397",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "compression_retriever = ContextualCompressionRetriever(\n",
    "    base_compressor=compressor,\n",
    "    base_retriever=vectordb.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "cde86848",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document 1:\n",
      "\n",
      "- \"those homeworks will be done in either MATLA B or in Octave\"\n",
      "- \"I know some people call it a free ve rsion of MATLAB\"\n",
      "- \"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data.\"\n",
      "- \"there's also a software package called Octave that you can download for free off the Internet.\"\n",
      "- \"it has somewhat fewer features than MATLAB, but it's free, and for the purposes of this class, it will work for just about everything.\"\n",
      "- \"once a colleague of mine at a different university, not at Stanford, actually teaches another machine learning course.\"\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Document 2:\n",
      "\n",
      "- \"those homeworks will be done in either MATLA B or in Octave\"\n",
      "- \"I know some people call it a free ve rsion of MATLAB\"\n",
      "- \"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data.\"\n",
      "- \"there's also a software package called Octave that you can download for free off the Internet.\"\n",
      "- \"it has somewhat fewer features than MATLAB, but it's free, and for the purposes of this class, it will work for just about everything.\"\n",
      "- \"once a colleague of mine at a different university, not at Stanford, actually teaches another machine learning course.\"\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Document 3:\n",
      "\n",
      "\"Oh, it was the MATLAB.\"\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Document 4:\n",
      "\n",
      "\"Oh, it was the MATLAB.\"\n"
     ]
    }
   ],
   "source": [
    "question = \"what did they say about matlab?\"\n",
    "compressed_docs = compression_retriever.get_relevant_documents(question)\n",
    "pretty_print_docs(compressed_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c4fc4d",
   "metadata": {},
   "source": [
    "## Combining various techniques"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "161ae1ad",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "compression_retriever = ContextualCompressionRetriever(\n",
    "    base_compressor=compressor,\n",
    "    base_retriever=vectordb.as_retriever(search_type = \"mmr\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e77ccae1",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document 1:\n",
      "\n",
      "- \"those homeworks will be done in either MATLA B or in Octave\"\n",
      "- \"I know some people call it a free ve rsion of MATLAB\"\n",
      "- \"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data.\"\n",
      "- \"there's also a software package called Octave that you can download for free off the Internet.\"\n",
      "- \"it has somewhat fewer features than MATLAB, but it's free, and for the purposes of this class, it will work for just about everything.\"\n",
      "- \"once a colleague of mine at a different university, not at Stanford, actually teaches another machine learning course.\"\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Document 2:\n",
      "\n",
      "\"Oh, it was the MATLAB.\"\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Document 3:\n",
      "\n",
      "- learning algorithms to teach a car how to drive at reasonably high speeds off roads avoiding obstacles.\n",
      "- that's a robot program med by PhD student Eva Roshen to teach a sort of somewhat strangely configured robot how to get on top of an obstacle, how to get over an obstacle.\n",
      "- So I think all of these are robots that I think are very difficult to hand-code a controller for by learning these sorts of learning algorithms.\n",
      "- Just a couple more last things, but let me just check what questions you have right now.\n",
      "- So if there are no questions, I'll just close with two reminders, which are after class today or as you start to talk with other people in this class, I just encourage you again to start to form project partners, to try to find project partners to do your project with.\n",
      "- And also, this is a good time to start forming study groups, so either talk to your friends or post in the newsgroup, but we just encourage you to try to start to do both of those today, okay? Form study groups, and try to find two other project partners.\n"
     ]
    }
   ],
   "source": [
    "question = \"what did they say about matlab?\"\n",
    "compressed_docs = compression_retriever.get_relevant_documents(question)\n",
    "pretty_print_docs(compressed_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c2b63e1",
   "metadata": {},
   "source": [
    "## Other types of retrieval\n",
    "\n",
    "It's worth noting that vectordb as not the only kind of tool to retrieve documents. \n",
    "\n",
    "The `LangChain` retriever abstraction includes other ways to retrieve documents, such as TF-IDF or SVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "83d2e808",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.retrievers import SVMRetriever\n",
    "from langchain.retrievers import TFIDFRetriever\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "bcf5b760",
   "metadata": {
    "height": 183,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load PDF\n",
    "loader = PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture01.pdf\")\n",
    "pages = loader.load()\n",
    "all_page_text=[p.page_content for p in pages]\n",
    "joined_page_text=\" \".join(all_page_text)\n",
    "\n",
    "# Split\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size = 1500,chunk_overlap = 150)\n",
    "splits = text_splitter.split_text(joined_page_text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "9bb0d781",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Retrieve\n",
    "svm_retriever = SVMRetriever.from_texts(splits,embedding)\n",
    "tfidf_retriever = TFIDFRetriever.from_texts(splits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0b1cc35f",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.9/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Document(page_content=\"let me just check what questions you have righ t now. So if there are no questions, I'll just \\nclose with two reminders, which are after class today or as you start to talk with other \\npeople in this class, I just encourage you again to start to form project partners, to try to \\nfind project partners to do your project with. And also, this is a good time to start forming \\nstudy groups, so either talk to your friends  or post in the newsgroup, but we just \\nencourage you to try to star t to do both of those today, okay? Form study groups, and try \\nto find two other project partners.  \\nSo thank you. I'm looking forward to teaching this class, and I'll see you in a couple of \\ndays.   [End of Audio]  \\nDuration: 69 minutes\", metadata={})"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"What are major topics for this class?\"\n",
    "docs_svm=svm_retriever.get_relevant_documents(question)\n",
    "docs_svm[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "2a1659c0",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=\"Saxena and Min Sun here did, wh ich is given an image like this, right? This is actually a \\npicture taken of the Stanford campus. You can apply that sort of cl ustering algorithm and \\ngroup the picture into regions. Let me actually blow that up so that you can see it more \\nclearly. Okay. So in the middle, you see the lines sort of groupi ng the image together, \\ngrouping the image into [inaudible] regions.  \\nAnd what Ashutosh and Min did was they then  applied the learning algorithm to say can \\nwe take this clustering and us e it to build a 3D model of the world? And so using the \\nclustering, they then had a lear ning algorithm try to learn what the 3D structure of the \\nworld looks like so that they could come up with a 3D model that you can sort of fly \\nthrough, okay? Although many people used to th ink it's not possible to take a single \\nimage and build a 3D model, but using a lear ning algorithm and that sort of clustering \\nalgorithm is the first step. They were able to.  \\nI'll just show you one more example. I like this  because it's a picture of Stanford with our \\nbeautiful Stanford campus. So again, taking th e same sort of clustering algorithms, taking \\nthe same sort of unsupervised learning algor ithm, you can group the pixels into different \\nregions. And using that as a pre-processing step, they eventually built this sort of 3D model of Stanford campus in a single picture.  You can sort of walk  into the ceiling, look\", metadata={})"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"what did they say about matlab?\"\n",
    "docs_tfidf=tfidf_retriever.get_relevant_documents(question)\n",
    "docs_tfidf[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7885389e",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
